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FOREWARD 


This  report  partially  fulfills  the  requirements  of  contract  N61339-85-D-0024,  D.0. 17  “Cost 
and  Training  Effectiveness  Analysis  Techniques,”  with  the  U.S.  Army  Simulation,  Training  and 
Instrumentation  Command  (STRICOM).  The  project  was  initiated  by  the  Training  and  Performance 
Data  Center  and  is  sponsored  by  the  Office  of  the  Assistant  Secretary  of  Defense  (Force  Management 
and  Personnel)  (OASD/FM&P).  Mr.  Don  Johnson  directs  the  project  for  OASD/FM&P. 

This  report  is  a  proceedings  of  a  symposium  held  at  the  14th  Annual  Meeting  of  the 
Interservice/Industry  Training  Systems  and  Education  Conference,  in  San  Antonio,  Texas,  Novem¬ 
ber  1992.  The  symposium  reported  herein  is  part  of  a  project  being  conducted  by  the  Institute  for 
Simulation  and  Training  of  the  University  of  Central  Florida  to:  (1)  assess  current  practices  by  the 
military  services  in  conducting  cost  and  training  effectiveness  analysis  (CTEA),  and  (2)  develop  a 
military  standard  for  conducting  CTEA. 

William  J.  Bramble,  Jr.,  John  Muller,  and  Valerie  Truhan  provided  considerable  help  in 
preparation  of  this  report. 


INTRODUCTION 
Michael  Parmentier 
Office  of  the  Assistant  Secretary  of  Defense, 
Force  Management  &  Personnel 


I’d  like  to  talk  about  some  major  points  that  lead 
the  military  to  where  we  are  today  in  assessing 
and  controlling  cost-effectiveness  of  training 
systems  and  to  to  give  you  some  background  as 
to  our  objectives  in  formulating  policy.  The  first 
pointisthattrainingisbigbusiness.  It  is  surprising 
the  amount  of  dollars  the  Department  of  Defense 
(DoD)  spends  on  training.  In  individual  training 
alone,  we’re  talking  in  the  neighboAood  of 
fifteen  to  twenty  billion  dollars  a  year.  Collective, 
or  unit,  training  brings  the  total  to  the  fifty  billion 
dollar  a  year  range.  Training  is  big  business  in 
defense  and  domestic  areas. 

The  second  point  is  that  military  training  is  the 
very  foundation  of  our  nation’s  continued 
strength  in  the  world.  The  utility  of  training  has 
been  proven  numerous  times,  the  most  recent 
demonstration  being  in  the  Desert  Storm/Desert 
Shield  campaign. 

We  all  know  where  the  budget  is  headed.  We  all 
know  where  the  forces  are  headed.  At  the  same 
time  as  these  things  are  headed  downward, 
technology  is  advancing  very  rapidly,  heading 
upward.  One  of  our  concerns  is  that  we  don’t  let 
technology  out-pace  our  ability  to  use  it  to  our 
advantage. 

Training,  in  many  cases,  is  high-risk  and 
uncertain.  Policy-makers  in  Washington  and 
from  services  throughout  the  country  see 
impressive  demonstrations,  and  their  response 
is,  “Gosh,  I  really  want  to  implement  that  stuff. 
How  do  I  do  it?’’  What  they  really  should  be 
asking  is:  “How  do  we  make  training  work  for 
us?  What  is  the  best  way  to  provide  training  to 
an  individual  or  a  group?  Should  we  use  a 
simulator,  a  computerized  model  or  an  actual 
operational  vehicle?” 


The  Role  of  the  Office  of  the  Assistant  Secretary 
of  Defense,  Force  Management  and  Personnel 

The  Office  of  the  Assistant  Secretary  of  Defense, 
Force  Management  and  Personnel  (OASD/ 
FM&P)  has  a  responsibility  to  monitor  the 
effectiveness  of  military  training  systems  within 
the  DoD  and  to  ensure  that  adequate  resources 
are  allocated  to  military  training  to  maintain 
combat  readiness. 

My  office  —  OASD/FM&P  (Readiness  and 
Training) — is  challenged  with  formulating  policy 
for  cost  and  training  effectiveness  of  military 
training  systems.  We  are  continually  responding 
to  fundamental  and  far  reaching  questions  relating 
to  training  budgets,  OPTEMPO  (such  as  fuel, 
ammunition  and  equipment  use),  large  training 
system  procurements,  and  the  relative  impacts 
on  readiness  of  competing  training  alternatives. 
As  the  DoD  budget  declines,  increasing  pressure 
will  be  placed  on  justifying  continued  training 
expenditures.  The  rationale  for  developing 
complex  training  systems  will,  of  necessity, 
include  documentation  of  their  cost  and  training 
effectiveness. 

Increased  guidance  and  oversight  by  the  Office 
of  the  Secretary  of  Defense  (OSD)  is  needed  to 
ensure  thatcost  and  training  effectiveness  studies 
are  systematically  conducted  and  documented. 
Methodologies  and  measures  of  effectiveness 
must  be  consistent  to  support  comparisons 
between  simulator-based  training  and  real  world 
training  exercises.  Results  of  analysis  must  be 
documented  and  utilized  by  decision  makers  at 
OSD  and  service  levels  to  support  future  training 
decisions. 


OASD/FM&P  responsibilities  for  overseeing 
Cost  and  Training  Effectiveness  Analysis 
(CTEA)  require  that  CTEAs  are: 

-  Based  on  mission  requirements 

-  Consistent  with  readiness  goals 

-  Consistent  with  service  mission 

-  Consistentwith  service  training  systems. 

Now  is  the  time  for  rethinking  the  issues  of  how 
we  are  going  to  use  simulators  and  models  as  we 
are  rethinking  issues  related  to  cost  and  training 
effectiveness.  Today,  we  have  some  experts 
who  are  going  to  discuss  relevant  work  that  is 
mostly  completed  and  address  where  we  need  to 
go  in  the  future.  They  are  going  to  need  your 
help  and  I  call  upon  you  to  provide  that  help. 

Introduction  of  Speakers 

I  will  now  briefly  introduce  each  of  the  panelists 
and  their  topic  areas. 

Peter  Kincaid  is  from  the  Institute  for  Simulation 
and  Training  at  the  University  of  Central  Florida. 
He  is  the  Principal  Investigator  for  the 
development  of  a  military  standard  for  cost  and 
training  effectiveness  being  sponsored  by  my 
office.  Thus  far,  the  study  team  has  evaluated 
and  documented  methodologies,  variables,  and 
measures  of  effectiveness  related  to  CTEA.  The 
emphasis  for  the  study  team  for  the  second  year 
of  the  project  (they  are  in  the  first  year,  now)  will 
be  to  develop  a  military  standard  for  conducting 
CTEAs. 

Michael  McCluskey  is  from  the  Army  Research 
Institute  (ARI)  Monterey  Field  Unit.  He  is  going 
to  address  work  related  to  unit  performance 
evaluation  at  the  Army  National  Training  Center 
(NIC).  The  ARI  Field  Unit  at  Monterey, 
California,  has  the  mission  of  storing  and 
analyzing  data  collected  from  the  NTC  and  other 
U.S.  Army  instrumented  ranges.  This  source  of 


data  represents  an  invaluable  asset  in  evaluating 
theeffectiveness  of  highly  realistic  field  training. 
Research  at  ARI  has  validated  principles  of 
training  which  have  been  demonstrated  to  have 
a  have  a  significant  effect  on  NTC  performance. 
For  example,  findings  have  related  the  NTC 
outcomes  to  various  aspects  of  training  at  home 
station,  including  OPTEMPO.  Evidence 
gathered  by  ARI  is  convincing  and  provides 
substantial  insight  into  principles  of  training  for 
optimal  transfer  to  operational  performance. 

Eduardo  Salas  is  from  the  Naval  Training 
Systems  Center  (NTSC)  in  Orlando,  Florida. 
His  work  focuses  on  team  training  issues.  The 
Naval  training  community,  as  well  as  the  training 
communities  of  the  other  services,  place  a  high 
priority  on  the  systematic  analysis,  design  and 
development  of  team  training  technology.  During 
the  past  few  years,  NTSC  has  been  conducting 
theoretically  based  and  practically  relevant  R&D 
to  enhance  our  understanding  of  what  team 
performance  is,  how  we  measure  it,  and  what 
instructional  strategies  can  best  be  used  to 
optimize  team  training.  Lessons  learned  from 
this  R&D  effort  have  substantial  implications 
for  the  design  of  team  training  systems,  for  the 
Navy  and  other  services.  The  R&D  is  continuing 
and  is  addressing  methodological  and  practical 
problems. 

Dee  Andrews  is  from  the  Air  Force  Armstrong 
Laboratory.  He  is  going  to  recap  work  some  of 
you  may  have  seen  earlier  in  the  week  on  the 
CTEA  for  the  C- 130  aircraft..  Then  he  is  going 
to  look  at  the  difference  between  cost  and  training 
effectiveness  and  cost-benefit  analysis.  He  is 
also  going  to  discuss  a  model  for  multi-ship 
training. 

The  US  Air  Force  has  adapted  the  policy  of 
converting  Aircrew  Training  Systems  (ATSs) 
from  primarily  Air  Force  operated  systems  to 
contractor  operated  systems,  specifically  the  ATS 
for  the  C-130.  Therefore,  it  is  vital  that  cost- 


effectiveness  data  be  collected  that  help  the 
government  better  understand  the  benefits  and 
limitations  of  contracted  ATSs.  The  C-130  is 
perhaps  the  largest  CTEA  ever  conducted  by 
any  U.S.  military  Service. 

The  cost-effectiveness  analysis  used  in  this  study 
provides  a  satisfactory  framework  for  evaluating 
changes  to  large,  complex  training  systems;  it 


may  even  serve  as  a  useful  model  for  the  other 
Services. 

Our  discussant  for  today  will  be  Jesse  Orlansky 
from  the  Institute  for  Defense  Analyses.  Dr. 
Orlansky  is  well  known  for  his  studies  related  to 
cost  and  effectiveness  of  military  trmning 
systems.  He  is  also  a  frequent  participant  in 
panels  such  as  this  one. 


Developing  a  Military  Standard  for  Cost  and  Training  Effectiveness  Analysis 

J.  Peter  Kincaid 
Institute  for  Simulation  and  Training 
University  of  Central  Florida 


The  principal  objective  of  our  project  which  I 
will  describe  to  you  today  is  to  support  the  Office 
of  the  Assistant  Secretary  of  Defense  for  Force 
Management  and  Personnel  (OS  AD/FM&P)  in 
developing  a  military  standard  for  conducting 
CTEA.  OSAD/FM&P  has  a  requirement  to 
monitor  the  effectiveness  of  military  training 
within  the  DoD.  Some  specific  tasks  that  must 
be  included  in  a  military  standard  are  to: 

•  Establish  procedures  and  policy  (perfor¬ 
mance-based) 

•  Monitor  effectiveness  of  military  training 
systems  within  DoD 

•  Ensure  allocation  of  adequate  resources  for 
military  training  to  maintain  combat  readi¬ 
ness 

The  major  purpose  of  military  training  is  to 
prepare  for  combat  and  other  operations;  yet 
training  is  not  often  specifically  evaluated  with 
this  goal  in  mind.  More  typically,  the  value  of 
training  is  measured  within  the  context  of  a 
training  event  (such  as  a  Tank  table  for  tank 
crews).  It  is  difficult  to  measure  the  effectiveness 
as  well  as  the  cost  of  a  fielded  training  system. 
Nevertheless,  knowledge  of  how  a  fielded 
training  system  provides  for  operational  readiness 
is  essential  for  designing  new  training  systems, 
which  almost  always  have  a  large  degree  of 
similarity  to  existing  systems. 

Decision  makers  in  the  military  services,  as  well 
as  in  the  Department  of  Defense  (DoD),  are 
regularly  faced  with  choosing  the  most  cost- 
effective  training  alternative.  In  this  era  of 
declining  military  budgets,  those  responsible  for 
procuring  expensive  training  systems  need  to 
justify  their  choices  with  validated  data  for  both 


cost  and  effectiveness  in  preparing  for  operational 
readiness.  Without  validated  cost  and  training 
effectiveness  data,  coupled  with  sound  analytical 
practices  for  choosing  among  training 
alternatives,  the  resources  available  for  training 
will  be  in  jeopardy.  Currently,  it  is  difficult  to 
make  valid  comparisons  among  competing 
training  requirements  as  practices  for  Cost  and 
Training  Effectiveness  Analysis  (CTEA)  are  not 
standard  across  services  and  military  agencies. 

TTie  intended  users  of  the  guidance  to  be  contained 
in  the  military  standard  are  government  decision 
makers  responsible  for  developing  and  acquiring 
training  systems.  These  decision  makers  are 
always  faced  with  budget  constraints  and  must 
find  efficient  alternatives  to  achieving  readiness 
requirements. 

Issues  Related  to  CTEA 

Here  are  a  number  of  quotes  from  papers  at 
meetings  such  as  this  one  and  contained  in  such 
“blue  ribbon”  reports  as  issued  by  the  Defense 
Science  Board. 

I  start  with  one  of  the  most  compelling  quotes, 
which  is  from  an  Army  Captain  in  Desert  Storm 
after  having  won  a  battle  fought  in  a  sandstorm 
in  which  his  forces  were  heavily  outnumbered. 
When  asked,  “Why  did  you  do  so  well?”  he 
replied,  “Sir,  this  isn’t  our  first  battle.  This  is  our 
tenth  battle.  We  fought  three  wars  at  the  National 
Training  Center  and  a  lot  of  other  simulations, 
like  SIMNET.”  Those  of  us  in  the  military  training 
community  love  to  hear  a  quote  like  that.  There 
was  general  consensus  that  part  of  the  success  of 
Desert  Storm  was  a  result  of  the  most  effective 
training  in  the  history  of  the  military. 


“Consideration  must  be  given  to  training 
effectiveness  during  the  design  and  development 
of  military  training  systems.  “ 

Weinclaw  &  Orlansky 
mEC.  1983 

This  is  a  quote  from  our  discussant  and  another 
author  at  this  meeting  nine  years  ago  bringing  up 
the  same  issue  that  we’re  dealing  with  today:  In 
fact.  Dr.  Orlansky  has  told  us,  that  this  issue  is 
one  that  he’s  been  working  on  for  some  thirty 
years  and  that  we’ re  still  addressing.  We  have  to 
address  it  now,  seriously. 

“Meaningful  and  timely  measures  of  the 
effectiveness  of  the  training  of  any  Army  unit  are 
tools  which  can  be  used  to  improve  the  training 
of  that  unit.„Currently  in  the  Army,  quantitative 
measures  relating  to  unit  training  are 
inadequate. " 

1985  Army  Science  Board 
Training  and  Training  Technology 

In  1985  when  this  statement  was  made, 
quantitative  measures  relating  to  unit  training 
were  in  considerable  need  of  improvement.  That 
situation  has  improved  considerably,  as  Mike 
McCluskey,  from  the  Army  Research  Institute, 
will  discuss  later.  His  Field  Unit  has  done  some 
really  excellent  work  at  the  National  Training 
Center  (NTC). 

Here  is  a  quote  from  PM  TRADE  (now 
STRICOM)  that  is  absolutely  true: 

“Wg  have  generally  been  unable  to  conduct 
hard  analyses  of  training  alternatives  because 
we  have  not  properly  measured  the  results  of 
training:  yet  despite  the  costs  of  training  systems, 
no  comprehensive  assessment  technology 
program  is  in  place.  ” 

1 989  PM  TRADE  Technology  Integration  Plan 


And,  the  most  recent  quote  comes  from  the 
current  Defense  Science  Board: 

“One  of  the  things  we  do  not  do  well  is  assess  the 
technicalfeasibility.  cost,  schedule,  and  military 
worth  of  systems  in  conceptformulation.  Military 
worth  relates  to  the  preparation  of  troops  for 
combat  readiness  and  is  the  most  important 
consideration. " 

1992  Defense  Science  Board 

Points  of  Impact  -  Training  System  Life  Cycle 

If  you’re  trying  to  make  an  impact  in  maximizing 
the  cost  and  training  effectiveness,  there  are 
several  points  where  you  can  do  it  We  use  a 
model  of  systems  development  which  is  shown 
in  Figure  1.  The  Defense  Science  Board  (1992) 
pointed  out  that  we’re  deficient  at  intervening  at 
the  concept  formulation  stage.  We  are  really 
much  better  at  intervening  in  the  design  and 
development  stage.  Most  of  the  tools  that  are 
used  by  the  military  services  and  contractors  in 
designing  training  systems,  and  in  designing 
weapons  systems  in  general  have  been  well 
developed,  and  used.  Once  the  basic 
configuration  of  the  system  has  been  set,  there  is 
much  less  opportunity  to  make  large  trade-offs 
that  can  result  in  either  maj  or  savings  or  increased 
efficiency  of  the  system.  Formal  evaluations  are 
done  at  deployment,  usually  the  first  field-testing. 
Some  systems  are  studied  very  effectively,  and 
some  systems  are  hardly  evaluated  at  all.  Once 
the  system  is  fielded,  there  are  operational  lessons 
learned  which  can  give  feedback  into  operational 
requirements.  So,  looking  at  this  simplified  life- 
cycle  of  the  training  system  (depicted  in  Figure 
1)  we  can  see  that  the  major  impact  will  be 
analyses  conducted  during  the  concept 
formulation  stage.  The  cost  issues  relating  to  the 
trade-off  for  training  effectiveness  and  similar 
analysis  have  to  be  done  during  the  entire  life- 
cycle.  We  also  need  to  address  these  issues  more 
systematically  than  how  it's  done  today. 


We’re  developing  policy  in  accordance  with  To  determine  service  CTEA  practices  and 
existing  standards,  such  as  MIL-STD-1379D  techniques,  we’ve  gone  through  many  abstracts, 
(Military  Training  Systems).  lmiddi\xon,SiSVirvey  most  of  those  have  come  from  the  Defense 
of  current  service  policy  for  cost  and  training  Technical  Information  Center,  over  2,000  of 
effectiveness,  conducted  at  1ST  has  found  them.  We’ve  identified  about  300  of  those  that 

substantial  inconsistencies  in  cost  and  training  we  consider  particularly  pertinent  to  our  effort 

effectiveness  data.  These  have  been  archived  in  a  database  called 

the  Training  Evaluation  Cataloging  System 
Theexpert  panel  with  whom  we’ve  been  working  (TECATS)  which  we  use  as  an  analysis  tool 

includes  members  from  the  Department  of  (DITRA,  1992).  We’renotparticularlyinterested 
Defense  Training  and  Performance  Data  Center,  in  establishing  a  database,  per  se,  but  rather 

now  DITRA  (Defense  Institute  for  Training  and  getting  a  snapshot  of  what’s  being  done  today 
Research  Analysis),  the  Army  Research  Institute,  and  some  insight  on  what  needs  to  be  done  to 

Army  TRADOC  which  includes  TRADOC  understand  current  cost  and  trainrng- 

AnalysisCommand  (TRAC),  the  Naval  Training  effectiveness  practice.  We’ve  had  two  expert 

Systems  Center,  Air  Force  Armstrong  panel  meetings.  One  was  held  in  Orlando,  and 
Laboratory,  the  Institute  for  Defense  Analyses,  one  was  hosted  by  the  Army  Research  Institute 
and  the  Institute  for  Simulation  and  Training  at  in  Monterey, 
the  University  of  Central  Rorida.  Resource 
Consultants,  Inc.  is  also  participating  in  that 
panel. 


POINTS  OF  IMPACT  - 
TRAINING  SYSTEM  LIFE  CYCLE 


Formal 

Analyses  Design  Guidelines  Evaluation 


Operational  Lessons  Learned 


Figure  1.  Points  of  Impact  for  Training  System  Life  Cycle 


Measures  of  Effectiveness 

Cost  and  training  effectiveness  policy  must  be 
based  on  a  foundation  of  operational 
requirements:  combat  readiness,  or  the  closest 
thing  to  combat  readiness  that  we  can  measure, 
such  as  nertormance  at  the  National  Training 
Center,  Red  Flag  and  major  fleet  deployments. 
Effectiveness  can  be  measured  on  a  number  of 
different  levels:  training  requirements,  objective 
data  validity,  training  strategy  (that  is  the  usage 
and  the  context  of  the  intervention,  for  example 
the  mix  of  the  use  of  operational  equipment  -  full 
mission  simulators,  inexpensive  computer-based 
training,  and  so  forth). 

Some  of  the  next  steps  will  be  developing  the 
Department  of  Defense  policy  guide,  based  on 
the  initial  findings.  We  will  be  establishing  a 
joint  working  group  of  practitioners.  These  will 
be  people  who  must  implement  the  military 
standard  once  it  is  promulgated.  We  will  be 
determining  some  of  the  commonalities  across 
the  services,  as  well  as  things  that  are  not  common 
across  the  services.  We  will  be  establishing 
measures  of  effectiveness  for  establishing  the 
military  standard. 

The  two  phase  study  is  outlined  below. 

Phase  1 

•  Conduct  literature  review  of  key  studies 

•  Document  significant  findings  in  PC-based 
cataloging  system 

•  Conduct  analyses  based  on  document 
findings 

Identify  existing  CTE  methodologies, 
variables  and  measures  of  effectiveness 
Document  trends  in  conduct  of  CTE 
studies 

•  Develop  matrix  of  requirements  vs.  training 
interventions 

Initial  matrix  is  for  Army 


Phase  2 

•  Establish  Joint  Service  program  elements 

•  Determine  common  Service  program 
elements 

•  Establish  Military  Standard  for  DoD  CTE 
Program 

This  concludes  my  presentation. 

Discussion 

AUDIENCE:  Do  you  intend  that  the  standard 
should  cover  cost  issues? 

KINCAID:  Yes,  that’s  our  intention  right  now. 
We  haven’t  examined  the  current  costing  figures 
as  much  as  we’ve  examined  other  things.  Part  of 
cost  and  training  effectiveness  is  cost;  we  have 
to  address  that. 

AUDIENCE:  Each  of  the  services  has  a  cost 
analysis  function  which  operates  according  to 
Service  guidelines.  What  are  you  doing  to  make 
your  guidance  consistent  with  service  standards? 
We  have  to  respond  to  that.  We  intend  not  to 
override  or  interfere  with  existing  procedures, 
but  to  establish  some  minimum  commonalities 
and  requirements  at  the  Department  of  Defense 
level,  and  then  to  examine  very  carefully  what’s 
already  in  place. 

If  you  in  fact  sponsor  a  training-effectiveness 
study  for  a  selective  training  system  acquisition, 
will  you  intend  to  do  this  in  parallel  with  the 
existing  service  requirements  or  do  you  intend 
that  as  a  substitution? 

KINCAID:  We’re  trying  to  piggy-back  on 
existing  Service  regulations  and  practices  as 
much  as  we  can.  We’d  like  to  do  this  without 
putting  new  requirements  on  the  services.  And 
indeed,  the  requirements  that  do  come  out  in  the 
military  standard  will  be  coordinated  with 
representatives  from  each  of  the  Services. 


PANEL:  Let  me  summarize  what  we  just  heard. 
For  those  of  you  in  the  audience  who  arc  working 
for  the  government  or  who  do  analytical  studies, 
one  issue  is  to  develop  methodology  providing 
information  relating  to  training  effectiveness. 
For  those  of  you  who  are  delivering  training 
programs  or  training  equipment,  your  attention 
ought  to  be  placed  on  the  means  to  support  those 


acquisitions  with  that  kind  of  methodology.  We 
need  to  remember  that  cost  is  for  the  life-cycle, 
and  training  effectiveness  should  relate  to 
readiness.  The  reason  to  make  that  point  is  that 
the  technology  to  evaluate  new  equipment  in  a 
combat  environment  is  becoming  available 
through  Distributed  InteractiveSimulation,  such 
as  SIMNET. 


Cost  /Training  Effectiveness  Analysis  and  Training  BeneHt 
Cost  Analysis  in  the  U.S.  Air  Force:  Two  Examples 
Michael  McCluskey 
Army  Research  Institute 
Monterey  Field  Unit 


The  mission  of  the  Monterey  Field  Unit  is  to 
conduct  research  to  develop  and  test  methods 
which  permit: 

Unit  performance  assessment. 

-  Field  validation  of  unit  training 
practices 

-  Development  of  methodology  to  sup¬ 
port  analysis  and  trends  of  Army  re¬ 
quirements,  unit  performance,  and  train¬ 
ing  practices 

and  to  increase  Army  unit  combat  performance 
capabilities  through  improved: 

-  Collective  training 

-  Management  of  collective  training 

-  Conceptualization  of  Army  require¬ 
ments. 

The  Field  Unit  Chief  is  Dr.  Howard  McFann, 
and  the  Field  Unit  is  part  of  the  Training  Systems 
Research  Division  which  has  its  headquarters  in 
Alexandria,  Virginia.  We  have  two  major  ac¬ 
tivities:  (1)  research  related  to  collective  training 
and  (2)  developing  techniques  for  unit  perfor¬ 
mance  measurement  An  additional  major  ac¬ 
tivity,  part  of  our  unit  performance  measure¬ 
ment  task,  is  to  maintain  the  Combat  Training 
Centers  (CTC)  Archive.  I  first  want  to  comment 
on  the  Arrr  y  requirement  for  the  research  and 
clarify  the  context  in  which  the  work  is  occur¬ 
ring.  The  requirement  placed  on  ARI  by  the 
Army  is  to:  ( 1 )  measure  unit  performance  effec¬ 
tiveness  at  the  CTCs  and  preparatory  conditions 
and  techniques  employed  at  home  station,  and 
(2)  quantify  the  relationship  associated  with 
effective  performance.  Our  objective  is  to  de¬ 


scribe  relationship  between  training  resource 
expenditures  at  home  station  and  unit  perfor¬ 
mance  at  the  CTC,  with  primary  emphasis  at  the 
National  Training  Center  (NTC).  The  approach 
we  arc  following  is  to  perform  a  longitudinal 
study  to  describe  the  relationship  between  train¬ 
ing  resource  expenditures  at  home  station  and 
unit  performance  at  the  CTC.  We  are  conduct¬ 
ing  a  two  phase  analysis  as  follows: 

-  Phase  1 :  Macro  analysis  of  extant  data 
Phase  2:  Detailed  micro  analysis  involv¬ 
ing  longitudinal  study  of  FORSCOM 
divisions. 

The  units  of  interest  arc  tactical  units,  both 
heavy  armor  and  light  mechanized  brigades, 
which  travel  to  NTC  to  experience  the  Army’s 
most  realistic  combat  training.  More  than  1,100 
vehicles  and  3,500  soldiers  are  transported  from 
their  home  station  in  the  continental  U.S.  for  a 
rotation  in  which  they  are  challenged  at  battalion 
task  force  level  to  fight  for  fourteen  days  and 
nights  in  some  of  the  harshest  terrain  in  the  U.S. 
Simulated  combat  situations  arise  both  in  force- 
on-force  and  live  fire  contests.  A  highly  trained 
and  dedicated  opposing  force  (OPFOR)  is  em¬ 
ployed  in  the  force-on-force  defense  and  offense 
exercises.  The  NTC’s  1,000  square  miles  per¬ 
mits  battalion  commanders  to  deploy  their  troops 
in  task  force  formations  and  undergo  some  six 
force-on- force  and  three  live  fire  day  and  night 
exercises.  InsU'umentation  provides  data  on 
such  factors  as  position  location  of  vehicles, 
firing,  and  communications  to  include  provision 
through  MILES  (Multiple  Laser  Engagement 
System)  to  record  hits  by  whom,  when  and  at 
what  range. 


Information  Sources 

We  receive  information  from  three  sources:  (1) 
the  NTC,  (2)  the  Joint  Readiness  Training  Cen¬ 
ter  (JRTC),  which  is  geared  for  Light  Infantry 
and  low  to  moderate  combat,  in  Arkansas,  and 
(3)  the  Combat  Maneuver  Training  Center 
(CMTC)  which  is  located  in  Germany. 

When  we  receive  information,  we  structure  this 
data  so  it  can  be  placed  in  the  archives  at  various 
places,  and  arrange  it  so  it  can  be  listed  in  a 
catalog  format.  The  catalog  is  automated  so 
analysts  can  find  the  information  they  need. 
Then,  after  workshops,  etc.,  we  get  feedback  on 
the  collection  process. 

Purposes  of  the  CTC  Archive  are  to: 

-  Maintain  the  Army  Archives  for  CTC 
data  and  records  and  mange  access 

-  Establish  and  maintain  a  research  data¬ 
base  for  measurement  research,  lessons 
learned,  and  trendline  analysis 

-  Maintain  a  research  data  base 

We  receive  a  wide  range  of  information  for  our 
CTC  Archive.  The  take-home  package  and 
after-action  reviews  are  the  primary  sources  of 
information  used  by  most  analysts.  They  con¬ 
tain  a  great  deal  of  training  assessment  informa¬ 
tion  organized  by  mission,  boss,  and  battle  phase. 
We  also  get  the  operation  orders  and  plans  and 
map  overlays  that  are  prepared  by  the  units.  In 
addition,  we  get  observer/controller  logs,  from 
individuals  who  are  controlling  the  events  that 
occurred  and  the  particular  conditions  that  ex¬ 
isted  during  that  rotation  exercises.  We  also 
receive  communications  tapes.  There  are  forty 
channel  communications  tapes  that  record  ev¬ 
erything  that  is  heard  during  the  exercise  pro¬ 
gram  by  the  rotating  unit  as  well  as  the  operating 
reserve  control.  We  have  firing  events  and 
casualty  information  that  we  receive  from  the 
observer/controllers,  either  from  the  MILES 
equipment  or  from  their  own  assessments. 


Purpose  for  Unit  Performance  Measurement 

There  are  three  main  purposes  for  unit  perfor¬ 
mance  measurement: 

1 .  To  develop  methods  to  provide  lessons  learned 
feedback  for  improving  Army  training  doctrine, 
material,  and  leadership. 

2.  To  develop  methods  to  provide  CTC  with  unit 
performance  information  for  feedback  as  part  of 
their  after  action  review  to  units. 

3.  Develop  methods  for  measuring  unit  combat 
performance  and  assessment  of  unit  combat 
readiness. 

Our  overall  research  strategy  is  to  “look  at”  units 
from  various  perspectives  which  include: 

INPUT  -  Input  is  the  type  of  effort  involved, 
such  as  delegating  equipment,  material,  fuel, 
ammunition,  training  facilities,  and  number  and 
quality  of  personnel. 

OUTPUT  -  Output  is  the  achievement  of  objec¬ 
tives  such  as  number  of  exercises  completed  or 
unit  proficiency. 

PROCESS  -  Process  includes  the  management 
and  training  techniques  used  to  obtain  quality  of 
performance  for  given  interrelated  inputs  such 
as  collective  tasks  in  the  Mission  Training  Plans 
(MTP)  and  NTC  performance. 

EFFECTIVENESS  -  Effectiveness  relates  the 
outputs  to  the  inputs  and  the  processes.  In  this 
area  researchers  will  examine  what  is  the  most 
efficient  way  to  use  input  and  processes  to  achieve 
desired  output. 

Home  Station  Training  and  NTC 

In  tests  that  we’ve  run  relating  home  station 
training  to  success  at  the  NTC,  a  number  of 
correlates  have  become  obvious  (Hiller,  McFann, 
and  Lenowicz,  1990;  McFann  and  McCluskey, 


1992).  In  terms  of  execution  in  training,  the 
brigades  that  did  the  best  followed  the  principles 
of  training;  train  as  a  team,  train  as  you  fight,  and 
use  performance  oriented  training.  Successful 
brigades  organized  early  for  combined  arms 
training  The  brigade  that  did  the  best  did  that 
well  in  advance  of  the  NTC  rotation — about  four 
months.  The  brigade  that  was  less  successful 
didn’t  do  that  until  about  one  month  before  the 
NTC  training.  As  to  “train  as  you  fight,”  the 
successful  brigadeemphasized  the  use  of  MILES, 
field  training,  and  combined  arms. 

Let’s  look  at  battle-staff  integration  in  terms  of 
NTC  performance.  We  have  the  percentage  of 
force-on-force  success  and  a  judgement  of  the 
degree  of  staff  integration  which  was  done  by 
the  commanding  staff  of  the  rotating  units.  Here 
we  can  see  the  importanceof  battle-staff  integra¬ 
tion  in  terms  of  the  success  of  the  success  of  the 
NTC. 

The  key  to  battle-staff  training  was  the  extent  to 
which  the  battle-staff  Standard  Operaitng  Pro¬ 
cedure  (SOP)  had  been  developed,  and  was  a 
part  of  the  combined  arms  training.  Again,  these 
are  readings  by  the  commander  of  the  staff 
concerning  the  extent  to  which  they  had  done 
that. 

The  realism  of  the  training  at  the  home  station 
was  another  factor.  For  the  most  successful 
brigades,  the  training  at  home  station  was  sup¬ 
posed  to  be  just  about  as  hard  as  the  ones  at  NTC. 
For  the  less  successful  brigades,  liieir  training 
was  considered  to  be  much  easier  than  at  NTC. 

Vehicle  operational  readiness  by  mission  se¬ 
quence  and  type  of  draw  was  another  big  factor. 
What  this  refers  to  is  the  percentage  of  vehicles 
that  were  operational  for  each  mission.  Now,  the 
type  of  draw,  in  the  physical  case,  refers  to  the 
case  where  the  unit  comes  to  the  NTC,  and 
accepts  the  equipment  given  to  them.  The  ex¬ 
tended  draw  refers  to  a  particular  unit  which 


came  much  earlier  to  the  NTC  and  spent  a  lot  of 
time  checking  out  the  equipment  and  making 
sure  it  was  operational.  If  not,  then  they  would 
make  sure  that  there  was  an  exchange  or  that  it 
was  made  operational.  Those  two  things  to¬ 
gether  led  to  big  differences  at  the  end  of  the 
rotation  in  terms  of  operational  equipment  We 
can  see  that  the  platoon  which  was  composed  of 
a  higher  percentage  of  less  experienced  soldiers 
did  not  perform  as  well. 

I  think  the  most  important  recommendation  re¬ 
fers  to  the  Mission  Essential  Task  List  (METL) 
process.  As  I  said  in  our  earlier  discussion  about 
the  training  management  cycle,  the  METL  pro¬ 
cess  needs  to  be  broken  in  terms  of  the  items  that 
are  put  on  the  METL  and  the  way  that  informa¬ 
tion  was  managed  from  that  point 

Now  I’d  like  to  review  a  list  of  the  characteristics 
of  the  most  successful  brigade.  The  most  suc¬ 
cessful  brigade:  had  a  higher  casualty-exchange 
ratio,  higher  training  miles  for  their  tanks  and 
Bradleys,  they  had  extensive  use  of  MILES, 
they  had  a  larger  number  of  field  days  on  discrete 
tasks,  they  had  the  highest  level  of  battle-staff 
integration  -  in  fact  they  trained  for  more  than  a 
year  together  on  battle-staff  integration.  The 
home  station  training  was  considered  to  be 
equivalent  to  the  NTC.  They  developed  battle- 
focus,  in  other  words,  specific  training.  There 
was  program  time  for  the  commander  to  assess 
and  repeat  training  as  they  went  through  the 
training  management  cycle  and  deficiencies  were 
found,  and  training  time  was  programmed  to 
make  corrections  for  them.  The  most  successful 
brigade  was  least  affected  by  training  distractors. 

Discussion 

AUDIENCE:  Have  you  done  any  work  relative 
to  the  number  of  hours  that  people  put  into 
simulation  in  trainers  and  how  that  affects  their 
performance  at  the  NTC. 


McCLUSKEY:  We  did  have  a  fair  amount  of 
data  that  was  collected  at  home  station  on  the  use 
of  training  aids  and  devices.  We  did  not  get  any 
relationship  partly  because  we  didn’t  have  good 
data.  What  we  did  get  was  informadon  about 
how  many  times  these  devices  were  checked 
out,  we  really  didn’t  home  in  on  effective  hours 
of  training.  We’re  going  to  have  to  do  a  more 
controlled  study  in  order  to  answer  that  question. 
I  think  it  would  be  safe  to  say  that  we  need  to  take 
a  look  at  the  relationship  between  hours  in  simu¬ 
lators  and  performance  at  the  NTC. 

PANEL:  The  services  have  a  priority  to  protect 
their  OPTEMPO.  In  some  cases  in  which 
OPTEMPO  has  been  reduced,  they  have  made 
certain  trade-offs  between  that  and  other  things 
they’ve  wanted  to  do.  There  is  no  strong  indica¬ 
tion  of  lower  levels  of  OPTEMPO  in  the  ser¬ 
vices.  The  Congress,  on  the  other  hand,  is  trying 
to  cut  OPTEMPO,  but  right  now  we  seem  to  be 
holding  our  own  pretty  well.  Historically,  there 
had  been  more  slack  in  budgets  so  that  if  you 
took  cuts,  you  could  make  it  out  in  other  areas. 
As  the  budget  draws  down  and  as  we  get  tighter 
on  resources,  I  think  that  the  services  are  going 
to  find  it  much  more  difficult  to  reallocate  funds 
from  other  areas  to  protect  the  OPTEMPO  that 
they  need.  So,  indeed,  it  is  going  to  be  a  lot 
tighter  than  it  was  in  the  past. 

AUDIENCE:  It  is  my  understanding  that  the 
Army  agreed  to  give  up  OPTEMPO  miles  for 
active  funds  in  exchange  for  funding  of  the 
Close  Combat  Tactical  Trainer  (CCTT). 

McCLUSKEY:  I’m  not  aware  of  theTRADOC 
issues,  as  you  mentioned  them.  Certainly  the 
relationship  between  OPTEMPO  at  home  sta¬ 
tion  and  performance  at  NTC  is  strong.  They’ve 
been  replicated  in  our  research.  Now,  how  we 
trade  off  OPTEMPO  with  training  devices  has 
yet  to  be  determined. 

PANELIST:  Let  me  make  a  general  comment 
about  that,  too.  I  can’t  speak  for  the  Army,  since 


I’m  not  in  the  Army.  What  you’re  going  to  see 
is  a  lot  of  focus  on  the  training  of  the  reserve 
components.  There  has  been  a  call  for  training 
of  the  reserve  components  at  the  centers  like  the 
NTC;  the  problem  is  availability  of  ranges,  etc. 
As  our  active  forces  reduce  size,  there  may  be 
more  space  for  the  reserve  forces  to  go  to  the 
NTC.  But  there  is  a  concerted  effort  on  the  part 
of  the  DoD  to  get  more  time  of  this  sort  for  the 
reserve  component.  It’s  just  going  to  be  a  matter 
of  time  before  this  works  out.  First  of  all,  you’ve 
got  to  figure  out  what  the  force  structure  is  and 
what  the  relationships  between  the  reserve  com¬ 
ponent  and  the  active  component  units  are.  Then, 
it  may  be  a  lot  easier  to  determine  how  you 
actually  invest  in  training. 

ORLANSKY:  Let  me  summarize  some  of  the 
issues  that  I  see  in  Mike’s  presentation.  I  think 
we  have  a  remarkable  relationship  (a  correlation 
of  .6)  between  tank  miles,  and  casualty  ex¬ 
change  ratio.  Its  fascinating  that  data,  as  limited 
and  as  unclear  as  that,  would  show  such  a  strong 
relationship.  It  does  not  follow,  however,  that  all 
the  groups  that  had  large  amounts  of  tank  miles 
performed  well.  The  issue  then,  ought  to  be  to 
look  for  the  ways  in  which  the  effective  groups 
use  their  tank  miles  as  distinct  from  those  that 
did  not.  Just  giving  everybody  tank  miles  would 
probably  be  a  silly  thing  to  do  without  assuring 
yourself  that  they  are  used  in  an  appropriate  way. 

Secondly,  a  good  deal  of  the  data  that  was 
described  here  was  gained  the  hard  way  -  after 
the  fact.  And,  again,  it  is  fascinating  that  they 
can  still  find  meaningful  relationships  with  that 
poor  of  a  research  design,  after  the  fact,  when 
data  were  originally  collected,  and  put  in  a  pile 
some  place  without  any  idea  of  who  would  want 
to  use  it  or  whether  it  was  even  the  right  data. 
Now  the  reason  that  I  make  this  point  is  that  in 
systems  like  SIMNET,  you  would  have  a  fantas¬ 
tic  capability  to  get  data  which  is  there  routinely 
all  of  the  time.  That  only  tells  you  what  hap¬ 
pened  during  those  combat  exercises,  but  that  is 


done  remarkably  well.  You  have  to  add  to  that 
all  of  the  training  plans,  and  the  command  staff 
relationships  which  the  SIMNET  data  don’t  use. 
The  data  issue  is  more  or  less  solved  in  systems 
like  SIMNET,  and  it  can  have  a  reasonably  long¬ 
term  view  in  the  ways  in  which  additional  data  of 
that  sort  can  be  melded  with  it  from  improve¬ 


ments  being  made  on  the  instrumented  at  ranges. 
You  will,  in  effect,  be  able  to  get  much  better 
range  data  in  the  future  that  will  actually  link 
very  closely  with  what  you’re  now  able  to  get  in 
SIMNET  and  in  simulators  now  under  develop¬ 
ment,  particularly  CCTT. 


Implications  of  Team  Training  R&D  for  CTEA 
Dr,  Eduardo  Salas 
Naval  Training  Systems  Center 


Those  who  study  organizational  effectiveness 
have  noted  that  teams  are  often  called  on  to 
perform  crucial  and  sometimes  hazardous  tasks. 
In  fact,  industrial,  governmental,  and  military 
organizations  are  especially  dependent  on 
successful  team  performance  in  order  to  ensure 
outcomes  such  as  competitiveness,  high  quality 
of  products  and  services,  readiness,  and  safety. 
Today  then,  it  is  imperative  that  the  science  of 
team  training  and  performance  helps  practitioners 
understand  the  dynamics  of  teamwork,  while  at 
the  same  time,  providing  guidance  for  optimizing 
team  ability  to  solve  organizational  problems. 
My  purpose  here  is  to  briefly  illustrate  what  our 
point  of  departure  was  in  approaching  the  issues 
associated  with  team  training  and  performance 
and  then  to  describe  how  this  relates  to  the  issues 
before  our  panel  today. 

Point  of  Departure 

Research  into  team  training  and  performance 
has  spanned  several  decades.  Many  diverse 
disciplines  have  devoted  time  and  resources  to 
team  performance  research,  and  numerous 
studies  of  team-group  performance  have  been 
reported  over  the  past  several  years.  Fields  such 
as  psychology  (military,  organizational,  social, 
and  engineering),  sociology,  instructional 
systems  development,  and  engineering  have 
sought  to  understand  the  nature  of  teams  and 
team  performance  and  how  to  best  design  team 
training  programs  and  systems.  Unfortunately, 
this  body  of  literature  has  yielded  few 
theoretically  driven,  empirically  based  guidelines 
or  prescriptions  for  designing,  managing,  and 
developing  teams.  In  short,  despite  over  50 
years  of  investigation  in  this  area,  we  are  only 
beginning  to  understand  the  complex  behavioral 
phenomena  associated  with  teams. 


Part  of  the  cause  of  this  state  of  affairs  was 
identified  in  the  mid- 1 980's.  Scientists  concluded 
then  that:  (a)  researchers  had  not  specified  or 
agreed  on  the  definition  of  a  team;  (b)  very  little 
systematic,  empirical  research  had  been 
conducted  that  addressed  the  process  (i.e., 
moment-to-moment  interactions)  of  team 
performance  to  show  what  fosters  or  hinders  the 
development  of  an  effective  team;  (c)  there  was 
a  lack  of  sound  performance  measurement  tools 
and  methodologies  for  capturing  the  dynamic 
and  often  time-dependent  nature  of  team 
performance;  (d)  the  knowledge,  skills,  and 
abilities  necessary  foreffective  team  performance 
needed  to  be  identified;  and  (e)  there  was  a  lack 
of  guidance  on  what  to  train,  when  to  train,  and 
what  instructional  environmental  demands  to 
reach  these  goals. 

From  a  behavioral  standpoint,  teamwork  appears 
to  comprise  a  complex  of  behaviors  including 
performance  monitoring,  performing  self¬ 
correction  of  errors,  providing  task  and 
motivational  reinforcement,  adapting  to 
unpredictable  situations,  using  closed-loop 
communication,  and  predicting  each  others 
behavior.  This  last  set  of  behaviors  is  intriguing 
in  that  we  can  now  use  concepts  (and  findings) 
from  cognitive  psychology  to  understand  team 
functioning.  For  example,  my  colleagues  Jan 
Cannon-Bowers  and  Shari  Converse  have 
hypothesized  that  team  decision-making 
performance  may  be  enhanced  via  training  to 
foster  "shared  mental  models"  of  the  task  and 
team.  That  is,  by  providing  team  members  with 
knowledge  (and  guided  practice)  about  their 
teammates,  task  requirements,  and 
interdependencies,  they  can  develop  accurate 
models  and  representations  of  the  team  task. 
Therefore,  team  members  would  learn  when, 
how,  and  why  to  pass  information  at  the 


appropriate  times,  without  the  need  to 
communicate  explicitly.  In  trying  to  uncover 
why  some  teams  work  well  and  others  do  not, 
our  measurement  methodologies  have  improved. 
My  colleagues  Dan  Dwyer  and  Cathy  Volpe 
have  developed  and  tested  tools  and  approaches 
for  capturing  individual  and  team  performance 
in  complex  environments.  The  methodology  is 
primarily  based  on  observation  of  key  constructs 
and  behaviors  for  each  team  member.  It  is  also 
tied  to  the  scenario  (i.e.,  simulation  exercise), 
creating  opportunities  to  observe  and  track 
individual  performance  as  well  as  team  processes 
outcomes.  Although  it  is  not  perfect,  we  can 
begin  to  diagnose,  remedy,  and  enhance  team 
performance  in  training  using  these  methods. 

We  know  from  our  experience  that  team  training 
and  performance  research  is  time  consuming, 
expensive,  labor  intensive,  and  slow  (i.e.,  results 
don't  happen  overnight).  Indeed,  our  ability  to 
conduct  meaningful  research  that  solves 
behavioral  phenomena  has  proved  to  be  a  major 
challenge.  We  have  concluded  that  in  order  to 
design  and  develop  instructional  strategies  for 
teams  in  naturalistic  settings,  a  research  program 
that  is  focused,  multifaceted,  long-term,  and 
resources  rich  (i.e.,  in  terms  of  people,  test-beds, 
and  funding)  is  required.  The  Tactical  Decision 
Making  Under  Stress  (TADMUS)  program  has 
been  our  attempt  to  develop  such  a  program. 

TADMUS  was  initiated  under  the  sponsorship 
of  the  Office  of  Naval  Technology  in  1990  in 
response  to  incidents  that  occurred  in  the  Persian 
Gulf  in  the  late  1980's.  The  goal  of  the  program 
is  to  gain  an  understanding  of  the  kinds  of 
decision-making  processes  that  occur  in  tactical 
command  and  control  teams  while  under  stress 
(e.g.,  heavy  workload,  time  pressure,  or 
ambiguity).  Based  on  this  understanding,  it  is 
our  intention  to  develop  training  and  simulation 
principles  (a  similar  component  that  focuses  on 
decision  support  systems  and  displays  is  being 
conducted  at  Naval  Command  and  Control  Ocean 


Surveillance  Center  in  San  Diego)  that  will  help 
to  mitigate  the  impact  of  stress  on  individual  and 
team  decision-making  performance. 

Where  Are  We  Going 

The  notion  of  shared  mental  models  (mentioned 
earlier)  is  one  of  the  theoretical  bases  on  which 
our  research  is  being  conducted,  see  ‘Individual 
Characteristics’  in  Figure  2.  Specifically,  the 
manner  in  which  we  hypothesize  that  shared 
mental  models  relate  to  team  effectiveness  has 
several  implications  for  the  understanding  of 
team  performance  and  training.  As  an 
explanatory  mechanism,  the  team  mental  model 
construct  is  useful  in  understanding  how  teams 
are  able  to  coordinate  behavior  and  select  task 
strategies  in  the  absence  of  explicit  coordination 
activities.  Under  conditions  of  high  workload, 
time  pressure  and  other  kinds  of  stress,  such 
implicit  coordination  appears  to  be  critical. 

With  respect  to  training,  strategies  designed  to 
foster  development  of  shared  mental  models 
have  the  potential  to  improve  team  performance. 
For  example,  we  are  investigating  the  following 
training  strategies. 

1.  Position  clarification.  Intervention  designed 
to  provide  information  regarding  the  structure  of 
the  team  and  task,  the  interrelationships  among 
team  member  positions,  and  the  roles  and 
responsibilities  of  each  team  member  could  be 
hypothesized  to  improve  team  performance  by 
enhancing  common  task  and  team  expectations. 
Such  training,  which  represents  requisite  team 
and  task  knowledge  could  be  presented  via 
information  presentation,  demonstration 
computer-assisted  instruction,  or  via  written 
material.  Such  training  would  represent  initial 
preparatory  training,  but  would  probably  not  be 
sufficient  to  develop  shared  mental  models. 
Another  potential  training  technique  that  may  be 
useful  for  this  purpose  is  role  playing,  which  also 
has  the  benefit  of  making  the  trainees  more 
active  participants  in  the  training. 


2.  Guided  practice  and  feedback.  Past  research 
has  indicated  that  unguided  practice  can  lead  to 
development  of  inaccurate  mental  models.  These 
findings  suggest  that  teams  should  practice  tasks 
under  the  guidance  of  instructors.  Feedback  and 
debriefing  mechanisms  must  be  designed  to 
result  in  accurate,  common  expectations  for  the 
task  and  team.  In  addition,  feedback  regarding 
specific  behaviors  that  must  be  changed  should 
be  more  effective  in  establishing  accurate 
expectations  than  general,  less  specified 
feedback.  Simulation  and  exercises  would  be  a 
most  appropriate  means  to  provide  such  practice 
opportunities. 


3.  Cross-training.  A  potentially  useful  strategy 
to  training  common  mental  models  may  be  to 
cross-train  team  members  on  tasks  that  are  related 
to  their  own  task.  Such  training  would  be 
beneficial  to  the  extent  that  it  helps  team  members 
to  learn  what  their  teammates  will  need  (in  terms 
of  resources,  information,  and  assistance),  given 
various  task  demands. 

4.  Team-leader  training.  Training  team  leaders 
to  foster  development  of  shared  mental  models 
also  has  potential  value.  It  can  be  hypothesized 
that  team  leaders  who  are  trained  to  articulate 
their  own  view  of  the  task  and  team,  who 


Figure  2,  Team  effectiveness  model  (Adapted  from  Salas,  Dickinson,  Converse  and  Tannenbaum) 


encourage  discussion  and  strategy  formation 
among  team  members,  and  who  make  clear  their 
expectations  of  team  members'  behavior  should 
be  successful  in  helping  their  teams  to  develop 
shared  mental  models. 

In  sum,  the  science  and  practice  of  team  training 
and  performance  is  maturing,  evolving,  and 
exciting.  The  1990's  look  promising!  We  will 
continue  to  further  our  understanding  of  this 
complex  behavioral  phenomenon  in  the  hope 
that,  through  theory,  methods,  and  data,  we  help 
organizations  solve  problems  in  the  future. 

Discussion 

PARMENTTER:  I  think  this  cost-benefit  analy¬ 
sis  issue  relates  to  the  Cost  and  Operational 
Effectiveness  Analysis  (COEA).  Apparently, 
with  the  development  of  the  new  acquisition 
regulations  (DoD's  “5000  series”),  the  Army 
was  faithfully  trying  to  conduct  COEA’s.  The 
other  services  were  less  faithful  with  regard  to 
responding  to  the  requirement  for  a  COEA.  I 
think  you’ll  find  that  the  lack  of  documentation, 
and  the  lack  of  formal  procedures  caused  some 
of  the  reluctance.  For  a  major  weapons  system, 
the  cost  of  doing  a  COEA  can  run  into  the  tens  of 
millions  of  dollars,  easily.  There  was  some 
resistance  to  accept  that  process. 

The  new  5000  series  regulations,  which  may  be 
old  before  too  long,  are  seriously  requiring 
COEAs.  In  the  new  acquisition  strategies  and 
processes  that  we’re  using,  the  other  services 
had  to  come  on  line.  There  was  some  discussion, 
informally  at  least,  by  the  Army,  as  to,  “Hey 
look,  we  keep  clocking  up  all  this  data  to  meet 
the  requirements  for  doing  the  COEA,  and  it’s 
being  used  against  us.  Those  other  guys  don’t  do 
it,  and  they’re  getting  away  with  it.  Let’s  re¬ 
think  this  process!  If  they’re  not  going  to  be 
required  to  do  it  too,  then  we’re  not  doing  it.” 
We  need  to  explain  to  everybody  how  they 
should  handle  this  in  a  uniform  fashion.  Hope¬ 


fully,  that’s  what  we’ll  be  able  to  achieve  when 
we  do  provide  the  guidelines  that  are  in  demand 
here. 

ORLANSKY;  I’d  like  to  comment  on  some 
very  important  changes  that  are  taking  place  that 
are  going  to  impact  on  military  training. 

There  is  a  requirement  that  new  developments 
in  technology  in  these  areas  be  demonstrated 
with  what’s  called  Advance  Technology  Dem¬ 
onstrations  using  simulation. 

The  performance  of  new  technology  will  be 
demonstrated  and  evaluated  in  a  simulated  envi¬ 
ronment,  of  precisely  the  sort  that  you  see  during 
this  convention.  One  of  the  major  areas  focuses 
particularly  on  man-in-the-loop  simulation  spe¬ 
cifically  synthetic  environments.  Distributed 
Interactive  Simulation  is  being  used  widely  as  a 
method  of  evaluating  the  effectiveness  of  new 
technology  in  these  key  high-priority  areas,  all 
of  which  involve  manned  operations.  That  is  the 
basic  theme  of  this  meeting. 

AUDIENCE:  Do  you  have  any  indication  of 
what  the  military  leaders  think  about  distributed 
simulation  and  the  utility  of  synthetic  environ¬ 
ments? 

ORLANSKY:  I  have  a  half-good  answer.  I 
know  as  a  fact  that  work  on  the  significance  of 
synthetic  environments  and  distributed  simula¬ 
tion  has  been  briefed  through  the  Joint  Chiefs  of 
Staff  (JCS)  and  the  Secretary  of  Defense.  I  don’t 
mean  that  they  were  overwhelmed  when  this 
was  described,  but  rather  that  changes  are  taking 
place  on  the  Joint  Staff  as  well  as  in  the  services, 
particularly  the  Army,  to  attempt  to  implement 
this.  That’s  the  thing  which  is  significant  A 
particular  office  has  been  assigned  responsibil¬ 
ity  for  running  this  program  in  the  JCS.  That 
office  did  not  exist  before.  Beyond  that,  it’s  not 
appropriate  to  say  what  the  JCS  will  do. 


But  significant  institutional  changes  are  being 
made  in  these  organizations  (as  reported  to  me 
by  a  senior  member  of  the  JCS).  That  looks  to 
me  like  a  very  positive  sign. 

I’d  like  to  focus  our  discussion  on  “What  do  we 
do  next?”  Given  the  well-developed  and  mean¬ 
ingful  background  that  has  been  discussed,  what 
should  we  do  to  cope  with  the  problems  in 
training  that  we  know  are  coming. 

One  key  issue  in  training  concerns  performance 
measures.  We’ve  done  quite  well,  I  think,  with 
measures  of  individual  performance  at  technical 
training  schools.  We’ve  done  modestly  well, 
but  not  impressively,  with  team  training.  But  I 
think  we’re  going  to  leap-frog  that  issue  of  team 
training.  I  doubt  that  we  will  ever  get  a  comfort¬ 
able  and  clear  answer  to  measuring  the  effective¬ 
ness  of  team  training.  Dr,  Salas  is  perfectly  free 
to  differ  with  me  on  that,  but  even  assuming  we 
know  how  to  do  that,  it’s  going  to  take  time  to 
develop.  But  that  does  not  really  bring  us  to  what 
we  really  want  to  know,  and  that  is  the  measure¬ 
ment  of  combat  effectiveness. 

Let  me  make  a  straightforward,  if  not  outrageous 
statement:  I  think  that  the  purpose  of  the  military 
is  to  win  wars.  If  you  believe  in  that,  then  you 
want  to  have  your  measures  of  performance 
related  as  well  as  you  can  to  how  well  it  helps 
those  folks  win  wars  on  our  behalf.  If  not  for  us 
as  people,  for  your  sons  or  daughters.  Take  this 
seriously;  it’s  an  interesting  and  important  per¬ 
spective.  The  question  about  military  training 
has  to  do  with  military  performance  and  with 
researchable  process  measures,  such  as,  for  ex¬ 
ample,  whether  people  talk  well  to  each  other, 
and  with  the  number  of  communications.  While 
these  are  probably  useful  process  measures  and 
you  couldn’t  perform  well  without  that,  the 
question  has  to  be  how  does  that  kind  of  perfor¬ 
mance  relate  to  performance  in  combat  on  the 
battlefield.  Those  are  measures  related,  essen¬ 
tially,  to  casualty  exchange  ratios  and  related 


items.  There  are  many,  many  measures  of  that 
sort  People  like  us  in  training  have  to  learn  how 
to  relate  the  effectiveness  of  performance  with 
what  it  would  mean  on  the  battlefield  against  a 
determined  enemy. 

I  don’t  know  why  this  point  has  to  be  made.  It  is 
one  thing  to  train  people  at  a  school  how  to  put 
together  and  take  apart  a  radar  at  school.  You 
wouldn’ t  have  an  effective  force  unless  the  radar 
worked,  so  that’s  important.  But  that’s  an  inter¬ 
mediate  measure.  What  you  want  to  get  out  of  all 
that  is  that  after  you  know  how  to  start  the  engine 
on  an  airplane,  turn  on  the  radar,  read  the  scope, 
and  talk  properly  on  the  radio,  you  want  to  know 
what  difference  it  makes  in  an  encounter  against 
a  determined  enemy. 

Our  concept  of  training  and  measurement  that  is 
relevant,  has  to  sooner  or  later  be  put  into  those 
terms.  That’s  not  as  hard  as  it  seems.  The  only 
thing  interesting  about  that  is  the  extent  to  which 
we  have  neglected  doing  the  obvious.  There  are 
other  kinds  of  folks,  such  as  the  military  and 
operations  research  folks,  who  do  this  all  the 
time.  We  ought  to  learn  how  to  listen  to  them. 
That’s  their  business!  When  you  use  a  combat 
model,  you’re  doing,  more  or  less,  what  we’re 
talking  about  in  training,  except  you’re  ducking 
theeffortit  takes  to  qualify  the  personnel.  You’re 
measuring  how  well  two  different  forces  might 
operate  against  each  other  as  estimated  by  a 
model.  That’s  a  substitute  for  what  would  hap¬ 
pen  in  a  training  exercise.  The  issue  still  is  at  the 
end:  how  does  one  force  operate  against  an¬ 
other?  Combat  models  are  based  upon  rules, 
algorithms,  combat  experience,  modest  guesses 
that  attempt  to  work  a  problem:  given  a  force  of 
a  particular  size  and  composition  with  certain 
weapons,  will  it  prevail  against  an  enemy  under 
certain  conditions?  If  you  think  that’s  idle  talk, 
that’s  the  way  in  which  many,  many  new  acqui¬ 
sitions  are  evaluated. 


We  have  an  enormous  opportunity.  Suddenly, 
we  have  the  capability  to  do  many  of  those  types 
of  analyses  on  a  simulated  and  controllable 
battlefield.  That  was  the  whole  point  of  showing 
you  where  the  technology  is  going,  and  where 
our  leadership  is  trying  to  move  us.  This  is 
toward  evaluating  equipment,  tactics  and  train¬ 
ing  exercise  environment  with  perform aiice  mea¬ 
sures  which  are  military  exchange  perforaiance 
measures. 

That’s  exactly  where  we  have  to  go.  We  have  to 
learn  what  others  have  done  with  this.  Then  we 
can  relate  it  to  our  own  analyses.  Regarding 
effectiveness  measures,  they  should  relate  to  an 
exercise  in  simulated  battle,  rather  than  profi¬ 
ciency  in  terms  of  curriculum  requirements  or 
other  training  measures.  In  simulated  battle,  it 
becomes  possible  to  collect  empirical  data. 

In  a  synthetic  environment  you  get  lots  of  perfor¬ 
mance  data,  whether  you  like  it  or  not.  These 
data  are  almost  more  than  we  know  how  to 
handle.  Though  there  is  an  issue  about  how  to 
use  it  well,  the  data  are  there.  It  is  difficult  to 
reconstruct  from  NTC  data  what  happened  in 
those  battles,  but  it  can  be  done  in  an  interesting 
way.  As  soon  as  any  SIMNET-type  exercise  is 
over,  the  data  are  there.  If  you  are  interested,  you 
can  even  review  the  data  while  the  exercise  is 
going  on. 

That  exists  now  in  SIMNET.  The  Joint  Tactical 
Exercise  Simulation  System  (JTESS)  will  give 
us  the  same  kind  of  data  on  an  enlarged  basis. 
JTESS  will  include  not  only  simulators,  but 
actual  weapons  on  instrumented  ranges,  and 
computer  models.  As  a  side  note,  this  is  the  same 
kind  of  equipment  that  we  would  use  in  actual 
combat.  The  equipment  is  now  not  very  differ¬ 
ent  from  what  you  could  get  during  an  actual 
exercise  or  during  a  war. 

So  the  question  of  “you  train  as  you  fight,  and 
you  fight  as  you  train”  is  going  to  happen.  What 
does  that  mean  for  us?  It  means  that  in  develop¬ 


ing  our  ways  of  evaluating  the  effectiveness  of 
training  equipment,  we  can  do  it  in  a  combat-like 
environment,  and  that  will  carry  over  to  an  actual 
combat  environment  In  different  ways,  I  stressed 
that  the  performance  measures  we  need  for  train¬ 
ing  ought  to  be  combat  related.  We  have  to 
figure  out  how  to  do  that  well. 

Finally,  there  is  one  more  point  that  I  would  like 
to  make  and  it  comes  out  of  the  Desert  Storm 
experience.  That  particular  war  was  a  fascinat¬ 
ing  case  for  America,  first  in  the  way  we  got  into 
it,  which  is  none  of  our  business,  and  then  the 
way  we  handled  it,  which  is  everybody’s  busi¬ 
ness.  I  heard  some  afternoon  reports  where  our 
military  leaders  made  a  very  good  point  It  was 
that  the  equipment  which  worked  very  well  in 
Desert  Storm  was  the  result  of  decisions  made 
fifteen  years  ago.  That’s  the  length  of  time  of  the 
development  and  acquisition  process.  If  we 
would  have  a  war  five  years  from  now  we  would 
use  equipment  that  exists  today. 

The  thing  that  was  special  about  Desert  Storm 
was  that  we  knew  the  enemy,  had  lots  of  training 
time  and,  on  the  political  side,  we  had  national 
concurrence.  Our  military  forces  were  able  to  do 
their  job.  They  knew  the  enemy.  I  think  the 
change  that  we  see  coming  in  may  not  give  us 
those  opportunities.  No  potential  enemy  will 
ever  be  that  stupid  to  allow  us  to  do  again  what 
we  were  able  to  do  in  Iraq.  In  the  future,  we  must 
be  able  to  engage  the  enemy  with  enormous 
force,  high  synchronization,  and  an  opportunity 
to  win  quickly  with  minimum  casualties.  If 
we’re  on  the  plus  side  of  this  scenario,  we  might 
engage.  If  we’re  on  the  minus  side,  the  chances 
are  that  we  would  avoid  getting  engaged. 

If  you  accept  this,  the  training  implications  are 
that  you  have  to  be  able  to  win  in  the  American 
way.  In  other  words,  we  need  synchronization, 
enormous  force  focused  for  short  periods  of 
time,  and  performance  that  will  minimize  casu¬ 
alties. 


Air  Force  CTEA  Programs 
Dee  H.  Andrews 
Aircrew  Training  Research  Division 
Air  Force  Armstrong  Laboratory 


Ill  begin  by  discussing  two  projects  performed 
at  the  Armstrong  Laboratory,  Air  Training 
Research  Division.  Our  research  concerns  pilots 
and  their  training.  In  conjunction  with  other 
groups  at  Armstrong  Laboratory,  a  contractor 
(SRA)  and  the  Military  AirLift  Command,  we 
have  recently  completed  a  very  extensive  cost 
and  training  effectiveness  study  with  C-130 
aircraft  (Derrick,  Tomczak,  NuUmeyer,  and 
Burright,  1990),  We  have  also  been  involved 
with  another  effort  to  develop  a  cost/benefit 
model  for  multi-ship  training.  Both  of  these 
projects  provide  tools  as  we  get  more  serious 
about  cost  and  training  effectiveness  analysis. 

In  my  opinion,  there  are  major  differences 
between  training  effectiveness,  cost  analysis  and 
cost/benefit  analysis.  With  training  cost- 
effectiveness  analysis,  the  basic  assumption  is 
the  cost,  or  the  quality  and  quantity  of  the  output 
(in  our  case,  graduates  from  the  training  program) , 
remains  constant.  The  benefit-cost  approach 
makes  some  assumptions  about  variability,  both 
quantity  and  quality  of  the  output,  and  cost  as 
well.  (Note:  see  the  discussion  at  the  end  of  this 
paper  relating  to  these  points). 

These  benefit-cost  studies  are  generally  much 
more  difficult  to  perform  because  we  have  to  do 
such  things  as  convert  the  benefit  into  a  metric 
which  can  be  compared  with  cost  (usually 
money).  Usually,  some  very  subjective 
judgements  must  be  made  about  benefits,  which 
is  not  easy  to  do, 

C-130  Training  Systems 

In  1982,  the  Air  Force  Scientific  Advisory  Board 
issued  some  recommendations  about  building  a 
model  aircrew  training  system.  The  model  was 


developed  by  1985.  The  first  attempt  to 
implement  the  model  was  a  contract  with  CAE- 
Link  to  convert  a  completely  blue-suit  training 
operation  in  the  C- 1 30  Aircrew  Training  System 
(ATS)  to  a  blue-suit/contractor  operation.  They 
tried  to  apply  guidelines  recommended  by  the 
Air  Force  Scientific  Advisory  Board  to  the  C- 
130  training  program.  This  was  generally  well 
done.  The  contractor  took  ground-based  training 
(academics  and  simulators).  The  blue-suit  folks 
still  do  the  flight  training. 

A  major  emphasis  of  the  new  aircrew  training 
system  was  a  shift  toward  lower  cost  devices. 
Computer-based  training  (CBT)  was  used  for 
the  first  time.  As  a  result  of  the  effort,  flight 
hours  were  reduced. 

The  development  of  the  new  aircrew  training 
system  was  a  massive  effort.  The  contractor 
examined  a  total  of  43  different  courses  involved 
with  the  C-130.  The  sheer  size  and  complexity 
of  the  program  and  interrelationship  between 
courses  made  it  difficult  to  do  a  benefit-cost 
study. 

Training  Cost-Effectiveness  Analysis 
Framework 

A  parametric  modeling  approach  was  set  up  to 
establish  cost-effectiveness  relationships 
between  several  important  variables.  We  spent 
considerable  effort  deciding  how  to  measure 
effectiveness.  We  finally  decided  that  check- 
rides  with  multiple  instructors  was  the  most 
important  and  best  measure  of  effectiveness. 

We  examined  four  solution  alternatives  which 
are  summarized  in  Table  1.  The  first  was  the 
Combat  Engagement  Trainer  (CET).  The  CET 


Table  1 

Training  System  Alternatives 


I.  Pair  of  GETS  in  each  squadron  -  linked  to  each  other  and  able  to  link  to  all  other 
sims  in  Wing 

II.  Pair  of  GETS  in  each  squadron  -  linked  able  to  link  to  all  other  sims  in 
DART  domes  in  Wing 

in.  GETs  in  each  squadron  -  linked  able  to  link  to  all  other  sims  in  wing  DART 
domes  in  Wing  and  other  Wings 

IV.  GETS  in  each  squadron  -  linked  ABle  to  link  to  all  other  sims  in  system  DART 
domes  in  Regional  Genters 


Table  2 

Four  Training  Alternatives  with  Gosts  and  Benefits 


Annual  Gosts  Annual  Benefits 


Alternative  I 

18  squadron  centers,  linked  $  12.9m  $32.6m 

Alternative  II 
18  squadron  centers,  linked 

to  other  squadrons  $13.3m  $46.9m 

Alternative  HI 
18  squadrons  centers,  linked 

plus  Wing  centers  $21.7m  $75.2m 


Alternative  IV 
18  squadron  centers,  linked 
plus  Regional  centers 


$29.6m 


$87.9m 


uses  a  glass  cockpit  and  costs  about  $300,000  to 
build.  It  is  fairly  high-fidelity  in  areas  such  as 
radar  representation.  The  CET  represents  the 
low-end  of  the  multiship  training  options.  We 
examined  pairs  of  CETs  in  each  squadron, 
throughout  all  the  F-15  squadrons  in  the  Air 
Force.  We  assumed  the  CETs  will  be  linked  to 
each  other  and  possibly  to  other  CETs  in  other 
squadrons.  Another  alternative  involved  looking 
at  a  pair  of  CETs  in  each  squadron,  linking  them 
to  each  other  and  to  the  others  in  the  wing,  using 
a  higher  fidelity  visual  device,  a  Display  for 
Advanced  Research  and  Training  (DART).  The 
DART  is  essentially  a  low-cost  dome.  For  the 
third  alternative,  we  used  CETs  in  each  squadron, 
which  are  then  able  to  link  to  all  other  sims  in 
Wing  and  in  other  Wings. 

Finally,  the  mostcostly  approach  is  similar  to  the 
three  others  except  that  we  use  regional  centers 
for  the  training  instead  of  doing  the  training  at 
the  units.  We  might  have  three  regional  centers, 
perhaps  two  in  the  United  States,  and  one  in 
Europe.  These  were  the  alternative  approaches 
we  considered  most  interesting. 

Table  2  shows  annual  costs  for  the  cost-benefit 
approach  described.  Annual  costs  include 
developmental,  research  and  other  costs 
presented  in  considerable  detail.  In  the  second 
alternative,  the  cost  varies  considerably  and  there 
is  less  benefit  than  for  some  of  the  other  options. 
However,  using  the  model,  if  you  look  at  the 
cost- ratio,  alternative  n  has  advantages,  primarily 
because  of  its  costs. 

When  cost-benefit  ratios  are  considered,  the 
specific  requirements  of  the  decision-maker, 
engineer,  squadron  or  wing  commander,  must 
be  understood.  This  is  a  very  subjective  process. 
By  looking  at  spreadsheets,  we  cannot  eliminate 
subjective  judgement,  it  is  still  very  much 
required.  A  benefit-cost  ratio  that  looks  very 
high  may  be  unacceptable  to  an  organization  for 
various  reasons.  But  we  do  have  sets  of 
spreadsheets  and  can  vary  any  of  the  costs  and 
related  benefits. 


Conclusions 

In  conclusion,  these  tools  are  very  useful  in 
assessing  cost  and  training  effectiveness.  The 
approach  that  the  contractor,  SRA,  developed 
are  quite  useful  when  it  comes  to  the  Air  Mobility 
Command  and  the  Armstrong  Laboratory.  We 
have  further  developed  this  benefit-cost  model. 
Other  benefit-cost  models  are  available,  but  for 
our  particular  requirements,  we  considered  it 
important  to  develop  our  own  model.  The  hardest 
part  involves  translating  benefits  into  a  dollar 
figure. 

The  personnel  who  perform  cost-training 
effectiveness  analyses  are  not  researchers.  They 
are  from  field  agencies  such  as  the  Army’s 
Training  Analysis  Center  and  the  Air  Force 
Operational  Test  and  Evaluation  Center 
(AFOTEC). 

As  I  understand  it,  AFOTEC  has  been  given  an 
expanded  mission.  They  are  responsible  for 
hundreds  of  new  projects.  Historically,  they 
have  looked  primarily  at  simulator  issues.  Now, 
they  are  getting  more  involved  with  training 
systems  evaluation.  They  are  trying  to  become 
more  adapted  to  the  notion  of  training  systems  in 
general  and  to  develop  expertise  in  that  area. 
They  should  be  provided  with  tools  that  will  help 
them  efficiently  conduct  training  effectiveness 
and  cost-benefit  studies.  As  mentioned  earlier, 
it  is  crucial  tliat  we  talk  to  decision-makers. 

The  approach  I  like  to  use  is  to  sit  down  with  a 
decision-maker  before  deciding  on  an  evaluation 
approach.  Present  the  decision-maker  with  two 
or  three  different  scenarios.  I  ask  the  decision¬ 
maker,  ‘What  decisions  would  you  make  if  we 
did  this  evaluation  and  found  X?  If  we  found  Y? 
If  we  found  Z?  Frequently,  that  cuts  directly  to 
the  concerns  of  the  decision-maker,  who  may  be 
unable  to  immediately  articulate  concerns.  In 
the  educational  literature,  there  are  numerous 
examples  of  decision-makers  who  do  not  know 


what  to  do  with  the  data  collected  after  costly 
evaluations.  It  is  crucial  for  those  who  collect 
the  data  to  meet  with  decision-makers  first  to 
determine  what  issues  are  of  greatest  concern. 

This  leads  to  the  next  point  Start  with  the 
leaders  at  the  top.  ‘Top  Brass”  in  the  Pentagon 
and  the  Congress  need  to  understand  the  meaning 
of  asking  for  a  particular  kind  of  data.  They  need 
to  understand  the  requirements  generated. 

The  final  point  is  that  efficiency,  in  my  opinion, 
is  the  heart  of  the  issue.  Efficiency  assumes 
effectiveness.  If  you  don’t  have  effectiveness, 
you  are  not  efficient.  You  are  just  cheap.  Cheap 
sometimes  is  not  sufficient  Y  ou  have  to  add  the 
extra  part,  the  effectiveness  issue. 

Discussion 

(Note:  Dr.  Burke  Burright,  an  economist  at  the 
Air  Force  Armstrong  Labs  was  asked  by  the 
panel  to  define  ‘‘cost-benefit  analysis.”) 

BURRIGHT:  Cost-benefit  analysis  really 
answers  two  related,  but  not  identical  questions. 
You  use  cost-effectiveness  analysis  to  identify 
the  most  efficient  way  of  reaching  your  stated 
goal.  Y ou  know  your  objective,  such  as  training 
some  individuals  to  increase  their  efficiency 
level.  Cost-effectiveness  analysis  is  used  to 
determine  the  most  efficient  way  of  reaching 
your  established  objective.  In  that  sense,  in  my 
way  of  thinking,  efficiency  and  cost-effectiveness 
are  synonymous. 

Cost-benefit  analysis  is  useful  when  you  are 
trying  to  evaluate  alternative  systems  which 
have  different  outputs  ai.  1  costs.  It  systematically 
evaluates,  in  dollar  terms,  the  present  values, 
and  the  alternative  system's  benefits  and  costs. 
The  difference  between  the  present  value  of  an 
alternative's  benefits  and  its  costs  is  its  net 
benefits.  To  decide  which  system  to  buy,  one 
should  look  at  the  alternatives'  net  benefits.  The 
one  with  the  largest  net  benefits  would  increase 
most  our  nation's  wealth. 


ANDREWS:  That's  where  I  think  our  model 
comes  in  so  handy.  Because  it  has  been  developed 
by  people  who  are  qualified  to  do  cost- 
effectiveness  analysis.  You  get  the  raw  input 
data  from  pilots  who  don't  know  cost- 
effectiveness  analysis  form  anything  else.  Once 
you've  got  their  estimate  of  benefits,  then  you 
can  bring  in  these  other  people  of  expertise  and 
their  approaches. 

ORLANSKY :  Are  there  models  so  elegant  that 
they  can  disguise  the  ignorance  with  which  you 
start?  I  think  the  issue  is  interesting  enough  to  be 
worth  a  debate,  and  you  ought  to  realize  that 
there  are  different  points  of  view  on  how  you  do 
an  analysis.  We're  trying  to  figure  out  a  way  to 
do  it  right,  rather  than  debate  it 

In  a  cost-effectiveness  evaluation,  you  have  two 
things  you  can  manipulate.  You  could,  in 
principle,  vary  effectiveness.  You  could,  in 
principle,  vary  cost.  But  that's  very  hard  to  do. 
So  all  of  the  analyses  are  truncated  to  specific 
eases.  Namely,  you  either  know,  or  you  assume 
that  the  output  or  performance  that  you  are  going 
to  get  with  a  new  system  is  about  the  same  as  you 
get  with  your  baseline  system.  For  practical 
purposes,  that  is  often  the  case.  It  is  not  a  bad 
assumption.  Then,  you  clearly  opt  for  the 
possibility  that  gives  you  the  same  result  at  less 
cost.  Y  ou  need  an  estimate  about  the  value  of  the 
benefits  in  dollar  terms.  It's  a  healthy  discussion 
to  acknowledge  that  you're  taking  those  estimates 
benefits  available  from  people  who  are  in  no 
position  whatsoever  to  know  what  kind  of 
judgement  they're  making  for  you. 

ANDREWS:  I'm  not  sure  that's  exacdy  true.  I 
mean,  these  are  people  who  fly  the  training 
sorties  in  the  aircraft,  they  do  know  something 
about  the  simulators.  When  you  ask  them  to 
compare  capability  or  potential  benefit  of  one  to 
another  you  may  have  some  problems,  but  I 
don't  think  their  opinion  is  inconsequential.  I 
value  their  opinion. 


BURRIGHT:  Icould  think  from  one  perspective, 
you  could  say,  “Why  ask  pilots?”  They  are 
amateurs.  What  do  they  know  about  this  stuff? 
On  the  other  hand,  my  question  is  -  who  knows 
better?  Who  do  you  view  as  experts  if  the  pilots 
are  not  allowed  to  speak  for  themselves? 

ORLANSKY;  The  point  I've  made  is  not  that 
you  wouldn't  want  to  do  that  kind  of  analysis. 
I'm  saying  we  don't  have  an  opportunity  and  the 
right  information  that  will  support  that  kind  of 
analysis.  Therefore,  you  don't  do  it.  Maybe  we 
can  come  back  to  this.  At  this  point,  let's  open  up 
for  general  discussion. 

AUDIENCE:  There  was  reference  made  to 
decision-makers.  The  target  for  a  study  of  this 
sort  would  be  some  decision-maker  at  the  con¬ 
gressional  or  Department  of  Defense  level. 
Would  decision-makers  lower  than  the  Depart¬ 
ment  of  Defense  or  Congress  use  this  type  of 
economic  analysis?  Who  do  you  want  to  read 
this  and  consider  these  alternatives? 

ANDREWS:  Those  who  worked  on  the  C- 130 
study  can  tell  you  that  they  have  been  briefing 
various  generals  in  the  Air  Mobility  Command. 
The  Air  Mobility  Command  decided  some  time 
ago  that  they  were  basically  going  to  contract  out 
virtually  all  their  ground-based  training  for  air¬ 
craft.  They’ve  done  more  than  just  the  C-130. 
So  this  is  the  first  time,  I  think,  that  anyone  has 
tried  to  capture  any  kind  of  cost-training  effec¬ 
tiveness  analysis  for  aircraft  switch-overs.  It 
looks  as  if  they  made  a  fairly  good  decision.  The 
output  remained  the  same  and  the  cost  is  prob¬ 
ably  less.  Then,  of  course,  they  want  to  extrapo¬ 
late,  I’m  sure,  to  the  other  aircraft  and  assume 
that  they  can  pat  themselves  on  the  back  and  say, 
“We  made  a  good  decision  about  those,  too.” 
However,  that  may  be  stretching  it.  It  may  be 
something  they  need  to  be  cautious  about. 

The  benefit-cost  model  I  mentioned  is  develop¬ 
ing.  Dr.  Orlansky  has  brought  up  some  very 
good  points  that  we  worried  about  from  the  start 


I  would  be  particularly  interested  in  people  such 
as  vendors  who  sell  whole  total  training  systems, 
the  F-22  for  example,  to  the  Air  Force.  This  kind 
of  a  model  might  be  of  some  use  to  them,  as  they 
are  trying  to  decide  what  to  propose,  and  how  to 
go  about  it  It  might  be  potentially  useful  to 
personnel  such  as  those  in  the  training  system 
procurement  office  up  at  Wright-Patterson  Air 
Force  Base.  They  have  to  make  decisions  about 
different  types  of  approaches.  It  could  even  be 
of  some  interest  to  the  folks  who  run  the  training 
centers  -  the  military  folks,  the  commanders,  or 
the  civilians  -  if  they’ve  got  to  make  some 
changes  to  their  training  systems. 

So  there  is  potential  there  but,  as  we  pointed  out, 
there  are  problems  as  well.  I’m  not  sure  all 
decision-makers  would  know  how  to  use  these 
types  of  analyses  or  would  want  to  use  them. 
There  are  other  ways  which  you  can  go  about 
making  these  decisions  which  they  do  all  the 
time. 

KINCAID:  A  general  way  of  asking  that  ques¬ 
tion  is:  what  good  is  a  detailed  analysis  to  a 
decision-maker  who  doesn’t  have  the  time  to 
really  study  it.  Typically,  the  decision-maker  is 
going  to  get  a  brief.  The  backup  data  is  available. 
It’s  done  by  the  military  agency  doing  the  analy¬ 
sis,  or  a  contractor  who’s  interested  in  how  a 
particular  piece  of  training  equipment  fits  into 
an  entire  training  strategy.  I  think  it’s  very 
valuable  that  a  briefing  has  some  validity  in 
terms  of  a  well  done  analysis  behind  it  that  may 
not  be  presented  in  detail  to  a  decision-maker. 

ORLANSKY:  The  C-130  study  is  absolutely 
first-rate.  The  issue  is  fairly  straight  forward. 
When  air  training  is  done  by  a  contractor,  the 
issue  for  the  government  is  whether  the  output  is 
the  same  as  when  performed  by  the  government 
itself.  The  government  took  the  position  that  it 
must  monitor  the  quality  of  the  output  For  the 
cost-effectiveness  study,  conditions  were  put  in 
the  contract  to  measure  the  nature  of  the  output 


and  to  permit  and  analysis  of  the  results.  A 
reasonable  assumption  was  made  that  the  output 
would  be  about  the  same.  Thus,  control  was 
placed  over  effectiveness  in  this  particular  case. 

Although  the  analytical  opportunity  is  about  as 
good  as  one  can  hope  for,  you  might  still  want  to 
be  a  little  bit  skeptical  about  the  clarity  of  the 
results.  The  skepticism  stems  from  the  variation 
in  “effectiveness  of  the  output”.  This  is  a  diffi¬ 
cult  measuring  issue.  In  the  military,  people  go 
out  of  their  way  to  help  those  who  don’t  proceed 
as  fast  as  others.  Thus,  it  isn’t  as  if  everybody 
gets  equal  amounts  of  resources  for  training. 
There  are  clearly  efforts  made  to  bring  along  the 
slow  ones.  There  is  disproportionate  support 
given  to  the  slow  ones  to  make  sure  they  can 
qualify.  But  that’s  not  bad.  I  think  that  working 
assumption  that  you  have  equal  effectiveness 
with  contractor  and  government  provided  train¬ 
ing  is  probably  a  good  enough  assumption. 

SALAS:  I'm  not  an  expert  on  cost  analysis  by 
any  means,  but  it  seems  to  me  that  most  of  this 


cost-benefit  analysis  is  done  for  political  rea¬ 
sons.  I  make  this  observation  because  it  seems 
that  we  spend  a  lot  of  time  looking  at  cost-benefit 
ratios,  and  other  detailed  information,  and  by  the 
time  you  raise  the  question  a  lot  of  the  decisions 
have  been  made. 

AUDIENCE:  The  cost-benefit  model  might 
work  best  in  concept  exploration,  whereas  some 
of  these  other  techniques,  tike  cost-effective¬ 
ness  come  in  later,  where  you  have  a  prototype. 

AUDIENCE:  After  doing  some  checking  around, 
I  found  out  that  the  Air  Force,  for  example,  does 
not  have  a  regulation  concerning  cost-training 
effectiveness  studies  for  training  systems.  When 
I  discovered  this,  I  was  surprised  because  of  the 
literally  hundreds  of  millions  of  dollars  that  have 
been  spent  on  training  systems.  I  know  of  Air 
Force  projects  which  have  used  an  Army  regula¬ 
tion  for  these  kinds  of  studies. 
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